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© A computer graphic interface (10) allows a user 
to obtain descriptive information concerning a fea- 
ture of a displayed image (21) by pointing to the 
location of the feature. Conversely, the user may 
enter descriptive textual information, and the loca- 
tions of responsive features are indicated. The data 
processing and memory storage requirements are 
minimized by encoding Information about the image 
as a pixel bit map (13), and a color map (25) in 
which the addresses or indices of the color map (25) 
are correlated with the addresses or pointers (27) to 
strings (26) of descriptive information. Each color 
map (25) address corresponds to a predefined set of 
features and descriptive information about those fea- 
i— tures. Since the pixel bit map (13) defines a color 
^map (25) address for each location on the image, 
suitable programming of the color map (25) can 
O insure proper correlation of descriptive information 
Owith corresponding locations on the image (21). The 
■^correlation between color map (25) addresses and 
00 the descriptive information about the features is re- 
W presented most compactly be arranging or sorting 
©trie entries in the color map (25) so that there is a 
correspondence between each predefined feature 
jjjand a continuous range of color map (25) addresses. 
Therefore, for a specified color map (25) address, 
the corresponding set of features and their pointers 



(27) can be found by comparing the specified color 
map (25) address to the limits of the color map (25) 
address ranges for the various features. 
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METHOD AND APPARATUS FOR IDENTIFYING FEATURES OF AN IMAGE ON A VIDEO DISPLAY 



BACKGROUND OF THE INVENTION 



1 . Field Of The Invention. 

The present invention relates generally to video 
displays, and more particularly to display systems 
in which an image is displayed to enable a user to 
recognize distinct features. Specifically, the present 
invention is directed to the problem of correlating 
the features of a displayed image with information 
which further identifies or describes the features. 



2. Description Of The Related Art. 

Systems for providing high-quality digital im- 
ages on video display screens are well known. The 
most common is known as a bit mapped raster 
scan display system. Such a system includes a 
memory which stores a set of bits corresponding to 
each pixel location on the display screen, and the 
information in the memory is read out and dis- 
played in synchronism with the raster scan. The 
memory including the set of bits for the pixels is 
known as the pixel bit map. 

For displaying high quality images having a 
variety of gray shades or colors, the set of bits for 
each pixel may specify the gray shade or color of 
the pixel. About 24 bits of resolution - eight bits for 
each of three primary colors — are required for 
reproducing the entire spectrum of chromaticity 
and luminance within the resolving ability of the 
human visual system. In order to reduce the mem- 
ory requirements for obtaining high color resolu- 
tion, it is well known to provide a color look-up 
table or "color map" which is indexed by the set of 
bits for each pixel. In this case the color map need 
store only once each color that is actually found in 
the image being displayed. 

The use of a color map has many advantages 
in addition to the reduction of the memory require- 
ments for the pixel bit map. As described in Hill 
U.S. Pat. 4,200,867, the color map can be used as 
a palette allowing the video display user to elec- 
tronically compose or "paint" video images. For 
mass storage of digital images or band limited 
transmission between video systems, the color 
map can be used to reduce the required bit trans- 
mission rate and bit storage capacity. In this re- 
gard, the color map can also be used in connection 
with other data compression techniques, such as in 
the color block coding scheme disclosed in Camp- 
bell etal., U.S. Pat. 4,580,134. 

The use of a color map further permits all 



pixels having the color identified by a single index 
to be modified simultaneously without disturbing 
the pixel bit map. The modification can be done for 
one color at a time by storing the color map in a 

5 read-write or random access memory, and writing a 
new color value to the memory address corre- 
sponding to the index of the color to be changed. 
Fleming et ai. U.S. Pat. 4,439,759, for example, 
describes the use of a data processor for providing 

70 color blinking by means of a linked list of multiple 
processes. For example, using this technique a bail 
may appear to bounce across an image, a river 
may appear to flow, or stars may appear to twinkle. 
As described further in Brown et al. U.S. Pat. 

75 4,484,187, it is possible Jo change all of the colors 
simultaneously by using a color map having a 
plurality of segments for the range of indices speci- 
fied by the pixel bit map, and by using a data 
processor to select which segment is to be ad- 

20 dressed by the pixel bit map. In other words, the 
data processor substitutes one color map segment 
for another color map segment in order to change 
simultaneously a plurality of colors. The memory 
capacity of the color map must be increased in this 

25 case, but in practice the substitution is easily per- 
formed by using a memory circuit having at least 
one additional address select line which receives 
an output of the data processor instead of the pixel 
bit map. Brown et al. also discloses that the sub- 

30 stitution of one properly programmed color map 
segment for another can instantly provide different 
priority for various images stored in different planes 
of bit map memory. 

In order to display moving objects which may 

35 obscure a background or obscure each other, it is 
common to provide a bit map memory which has, 
for each pixel, a respective set of bits for each 
object or the background. Each set of respective 
bits is stored in a respective "bit plane" for the bit 

ao map memory. The respective sets of bits for each 
pixel are compared to each other in a priority 
decoder which has an output which controls the 
select input of a multiplexer which selects one of 
the sets of bits for display. The priority decoder, for 

45 example, operates the multiplexer to select the set 
of bits which represents the largest binary number, 
or selects the set of bits which represents a non- 
zero binary number from the bit plane assigned the 
highest priority. In the latter case, the binary num- 

50 ber zero represents a transparent area in the bit 
plane. See, for example, Ciarcia, "High-Resolution 
Sprite-Oriented Color Graphics," Byte, August 
1982, pp. 57-80. 

The priority of an object can also be encoded 
as an attribute separate from the color map index. 
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Peterson U.S. Pat. 4,675,666, for example, de- 
scribes a video display system in which a user can 
display "true" objects by individual pixels in a "bit- 
plane mode," or fixed objects in a "list mode." The 
fixed objects in a list mode are defined in "image 
tables" which contain their pixel-by-pixel descrip- 
tion. A list mode screen memory is arranged as a 
"display list" of pointers to entries in the image 
tables, which include "attributes" of the images. 
These attributes allow each individual occurrence 
of an image to be altered, for example, by underlin- 
ing or flashing. By using a suitable attribute list and 
a suitable priority decoding circuit, Peterson says 
that the color of a "true" object overlapping a fixed 
object of lesser priority may be modified such that 
it appears that the true object has entered the 
shadow of the fixed object. 

Brown et al. U.S. Pat. 4,484,187 discloses that 
the priority decoding function can be fully program- 
mable by providing a unique color map address for 
each combination of bits from the respective sets 
of bits from the various bit map planes. Specifi- 
cally, the output from the memory for one bit plane 
is used as one portion of the color map address 
and the output from the memory for another bit 
plane is used as another portion of the color map 
address. Therefore, the collisions or overlap be- 
tween objects in different bit planes provide unique 
color map addresses, and the color map can be 
programmed at these addresses to provide any 
desired color for the points of collisions or overlap. 
FIG. 6 in Brown et al. shows ranges of color map 
addresses assigned to respective objects in a spe- 
cific example. 

A color map display system allowing rapid 
shifting of subpictures and rapid changing of the 
subpictures or the overall picture (such as having 
invisible information suddenly appear on the dis- 
play screen) is described in Allen et a!. U.S. Pat. 
4,570,217. As described in Col. 4, line 50 to Col. 5, 
line 17, the screen is broken down into a plurality 
of zones, each zone providing up to sixteen dif- 
ferent colors selectable for each pixel in the zone. 
The 16 colors from each zone are selected from 
one of four color palettes. Each color palette in turn 
selects its colors from up to 512 separate colors. 
Typically, a zone comprises eighty pixels of graph- 
ic information and thus each of those eighty pixels 
can be selected to have any one of the zone 
colors. It is said that by use of the color palette 
technique in association with each of the plurality 
of zones, the graphics can present complicated 
displays. Also, a high level graphics language is 
described for facilitating the design and configura- 
tion of a process control system. The graphics 
language has static and dynamic commands for 
facilitating graphics display update on a real time 
basis. 



The ability to recognize colors in a real-world 
digital video image is important for geological and 
agricultural assessments, military reconnaissance, 
city planning and land use, and coastal oceanog- 

5 raphy. Dalke et al. U.S. Pat. 4,488,245 discloses a 
computer and video display system which is said 
to be suitable for such color recognition applica- 
tions. The color picture is digitized and represented 
as three eight-bit words for each pixel in a 1024 X 

w 1024 pixel array image. The first value is an ap- 
proximation of luminance, while the second and 
third values represent variations of chromattcity or 
chrominance. Pattern recognition techniques are 
used to determine if a given color value lies within 

is a specified three-dimensional color sub-space. A 
high-speed digital processing technique is said to 
enable the real time modification of any portion of 
the color image within the display. A weighting 
function is disclosed to enable the operator to 

20 modify or insert colors. 

In Col. 15 lines 12-21, it is said that in opera- 
tion a threshold circuit is used to generate a mask 
to enable the operator to determine if, in fact, a 
color selection circuit has correctly identified all of 

25 the objects having the targeted colors; the mask is 
automatically superimposed over all of the areas on 
the color display having the same color value as 
the pixels selected during the sampling process. In 
Col. 15, lines 55 to 58, it is said that if desired, 

30 additional circuitry may be incorporated in the data 
processing unit to enable the operator to identify 
certain areas by their physical coordinates using a 
cursor, and a light pen or stylus. 

In simulation programs using the techniques of 

35 artificial intelligence, it has become common to 
provide a user interface which displays a sche- 
matic diagram of a system under, consideration, 
and which permits the user to graphically select or 
modify components of the schematic diagram by 

40 manipulating a pointing device. Further, upon such 
selection or modification, certain textual attributes 
of the object will be automatically modified if nec- 
essary and displayed in a window area adjacent to 
the schematic diagram. In such programs, the co- 

45 ordinates of the cursor are compared to coordi- 
nates defined for the components of the schematic 
diagram, in order to determine whether the user 
has selected a component. See, for example, Bor- 
ning, "Thinglab - A Constraint-Oriented Simulation 

50 Laboratory", Chapter 2, Stanford Computer Sci- 
ence Department Report STAN-CS-79-746 pp. 14- 
37 (July 1979). 



55 SUMMARY OF THE INVENTION 

The primary object of the present invention is 
to provide an improved method of identifying 
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predefined features of an image on a video display. 

A specific object of the invention is to provide 
an improved method of operating a video display 
system for providing descriptive information re- 
sponsive to user identification of the physical loca- 
tion of a perceived feature of a displayed image. A 
related object is to provide such a system with the 
capability of receiving a description from the user, 
and indicating the physical locations of predefined 
features which are responsive to the description. 

Another object of the invention is to provide an 
efficient method of storing digitized video images 
along with descriptive information about various 
features of the images and information about the 
locations of the features in the images. 

Briefly, the advantages of the present invention 
stem from encoding information about a video im- 
age as a pixel bit map and a color map in which 
the addresses or indices of the color map are 
correlated with the addresses or pointers to strings 
of descriptive information about predefined features 
of the video image. In other words, each color map 
address corresponds to a predefined set of fea- 
tures and descriptive information about those fea- 
tures. Since the pixel bit map defines a color map 
address for each physical location on the image, 
suitable programming of the color map can insure 
proper correlation of descriptive information with 
corresponding physical locations on the image. The 
correlation between color map addresses and the 
descriptive information about the features is repre- 
sented most compactly by arranging or sorting the 
entries in the color map so that there is a cor- 
respondence between each predefined feature and 
a continuous range of color map addresses. There- 
fore, for a user specified physical location, the 
corresponding color map address can be found by 
addressing the pixel bit map. and the correspond- 
ing set of features and pointers can be found by 
comparing the corresponding color map address to 
the limits of the color map address ranges for the 
various features, for example, by a binary tree 
search or by a table look-up procedure. 

Conversely, given a user specified description, 
the descriptive information for the various features 
can be searched to determine whether there is a 
correspondence with the user specified description, 
in order to find the features responsive to the user 
specified description. Next the address limits of the 
color map memory are found which correspond to 
responsive features. The physical locations of the 
features on the displayed image are then indicated, 
for example, by temporary modification of the color 
map entries at each address within the address 
limits of the responsive features. 

It should be apparent that separate pixel bit 
maps are not required for indexing both the color 
map and a pointer file in order to correlate each 



pixel with one or more strings of descriptive in- 
formation. Moreover, in most real-world applica- 
tions, the features of interest are highly correlated 
with ranges of color in the real-world images. 

5 Therefore, it is possible to use a single pixel bit 
map or pixel plane of common indices for repre- 
senting the video image and also for correlating 
descriptive information with the pixels, without re- 
quiring the memory capacity of the color map to be 

io substantially increased, and without loss of per- 
ceived image quality. By assigning or sorting the 
entries of the color map so that predefined features 
correspond to respective continuous ranges of col- 
or map addresses, the memory space for storing 

is the pointers is reduced. Therefore, digitized video 
images are capable of being efficiently stored 
along with descriptive information about various 
features of the images and information about the 
locations of the features' in the images. 

20 

BRIEF DESCRIPTION OF THE DRAWINGS 

Other objects and advantages of the invention 
25 will become apparent upon reading the following 
detailed description and upon reference to the 
drawings in which: 

FIG. 1 is a schematic diagram of a video 
display system incorporating the present invention; 
30 FIG. 2 is a diagram illustrating the organiza- 

tion and contents of a color map, a text pointer 
table, and certain text strings used to represent and 
describe the image and various features of the 
image displayed as shown in FIG. 1; 
35 FIG. 3 is a flowchart of the procedure ex- 

ecuted by a data processing unit in the system of 
FIG. 1 ; and 

FIG. 4 is flowchart of the procedure ex- 
ecuted by the data processing unit for encoding 
40 images and textual information describing their fea- 
tures in the format shown in FIG. 2. 

While the invention is susceptible to various 
modifications and alternative forms, a specific em- 
bodiment thereof has been shown by way of exam- 
45 pie in the drawings and will herein be described in 
detail. It should be understood, however, that it is 
not intended to limit the invention to the particular 
form disclosed, but on the contrary, the intention is 
to cover all modifications, equivalents, and alter- 
so natives falling within the spirit and scope of the 
invention as defined by the appended claims. 



DESCRIPTION OF THE PREFERRED EMBODI- 
55 MENT 

Turning now to FIG. 1, there is shown a sche- 
matic diagram generally designated 10 of a video 
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display system incorporating the present invention. 
As is conventional, the video display system in- 
cludes a color video display 11 for displaying text 
and images. The video display preferably includes 
a raster-scanned color cathode ray tube as a dis- 
play device, although the invention is applicable to 
other video display devices such as liquid crystal, 
plasma, and electro-luminescent displays, and the 
invention is also applicable to other kinds of scan- 
ning methods. A most suitable display device is, 
for example, an RGB analog video monitor sold by 
Sony Corporation. 

In addition to the cathode ray tube and its 
analog circuits, the video display 1 1 includes raster 
scanning circuits 12 which generate pixel address- 
es in synchronism with the scanning of the cathode 
ray tube, a pixel bit map or memory 13 which 
stores information about the image intensity to be 
generated by the cathode ray tube at each pixel, 
and a digital-to-analog converter 14 responsive to 
video data for generating analog signals which con- 
trol the electron guns in the cathode ray tube. 
Conventionally, the raster-scan circuits 12, the pixel 
bit map 13, and the digital-to-analog converters 14 
are included on circuits that are in close proximity 
to a data processing unit 15 such as a micropro- 
cessor, and a certain amount of random access 
memory 16 which is connected to the data pro- 
cessing unit via address and data buses. The pixel 
bit map 13 is typically included in a memory that is 
multiplexed or time shared between the raster- 
scanning circuits 12 and the data processing unit 
15. Typically the data processing unit 15 changes 
the data in the pixel bit map 13 during the vertical 
retrace of the raster-scanning, so that the entire 
image displayed by the display 11 appears to 
change instantaneously. 

For permanently storing large amounts of data, 
and also for providing input of large amounts of 
data, the video display system 10 includes disc 
memory 17 such as a floppy magnetic disc drive, a 
hard magnetic disc drive, or an optical disc such as 
a CD-ROM drive. 

For receiving text input from a user, the video 
display system 10 has a keyboard 18 which is 
connected to the data processing unit 15 through 
an interface circuit 1 9. 

The video display system 10 further includes a 
manually-operated pointing device 20 for receiving 
an indication of a specified location on the image 
21 displayed by the video display 11. As shown in 
FIG. 1, the pointing device is a "mouse" which the 
user manually rolls over a flat surface 22. The data 
processing unit 15 is conventionally programmed 
to display a cursor 23 at a particular pixel on the 
image 21, and the data processing unit is further 
programmed to move the cursor up and down or 
left and right in response to the user's movement 



of the mouse 20 forward and back or left to right. 
The user moves the mouse 20 until the cursor 23 
is aligned with a desired pixel location, and the 
user then specifies the desired pixel location by 

5 depressing a push button switch 24 on the mouse 
20. It should be understood, however, that the kind 
of pointing device used by the operator is of no 
consequence to the practicing of the present inven- 
tion, since a wide variety of other kinds of pointing 

10 devices can be used, such as a "track ball," a "joy 
stick,," or a "light pen." All of these devices enable 
the user to specify a desired location on the image 
21 displayed by the video display 1 1 . 

As described above, the video display system 

75 uses components which are found in a wide variety 
of programmable video display terminals and 
microcomputers. For practicing the invention, the 
video display system 10 further includes a color 
map 25 which is used in conjunction with the pixel 

20 bit map 13 to specify video data fed to the digital- 
to-analog converter 14 of the video display system. 
Color maps have provided a wide variety of advan- 
tages in video display terminals, as should be 
apparent from the detailed discussion above of the 

25 background of the present invention, in short, the 
color map 25 is used to represent the colors in the 
image independent of where those colors appear in 
the image. Specifically, the color map is a kind of 
memory which specifies the video data for each of 

30 a plurality of indices, and the pixel bit map 13 
specifies one of these indices for each of the pixels 
in the image 21. 

Suitable raster-scan circuits 12, pixel bit map 
13, color map 25, and digital-to-analog converters 

35 14 are found in commercially available video 
graphic circuit boards for personal computers. For 
practicing the invention, the inventor has used a 
"#9 Revolution Board" manufactured and sold by 
the #9 Computer Corp., 725 Concord Avenue, 

40 Cambridge, Massachusetts. This video graphics 
circuit board was inserted in an iBM-XT personal 
computer. Such a video graphics circuit board has 
a pixel bit map with 8 bit words for storing an index 
ranging from 0 to 255, and sufficient memory ca- 

45 pacity for a pixel matrix array of 512 columns by 
484 rows comprising the image 21 . Moreover, the 
color map 25 specifies 24 bits of video data for 
each index, including 3 bytes of 8 bits each which 
specify the intensity of the red, green and blue 

so primary colors. 

The video display system of the present inven- 
tion allows the user to obtain descriptive textual 
information concerning a feature of the displayed 
image 21 by using the pointing device 20 to point 

55 to a location of the feature. Conversely, the user 
may use the keyboard 18 to enter descriptive tex- 
tual information, and the locations of responsive 
features are indicated by the video display 11. As 
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shown in FIG. 1, the image 21 includes four distinct 
features which are recognized as a chair, a lamp, a 
sofa, and a table. It should be readily appreciated, 
however, that the present invention has great prac- 
tical utility in connection with the identification of 
features occurring in complex real-world images 
such as the images that are analyzed for military 
reconnaissance or the remote satellite sensing of 
agricultural crops or land areas for mineral explora- 
tion. The inventor has found the present invention 
especially useful for identifying predetermined fea- 
tures of medical images and, in particular, tissue 
cross-sections. A collection of tissue cross-section 
images in the form of a "histology atlas" would be 
of great benefit to doctors and medical students as 
an aid in teaching and diagnosis. 

Turning now to FIG. 2, there is shown a sche- 
matic diagram illustrating a specific organization of 
the color map 25 and certain data structures in the 
random access memory 16 which illustrate the 
method of the present invention. 

In accordance with an important aspect of the 
invention, the color map 25 includes at least one 
entry for each different color included in the image 
21, and includes additional entries for each color 
which occurs both inside and outside a predefined 
feature of the image. Therefore, a unique entry in 
the color map is provided for each color included 
in a distinct combination of features. Therefore, 
each index has associated with it a predefined set 
of features and a certain color. 

For the predefined features in the image 21 
shown in FIG. 1, there is no overlap between the 
features. Therefore, the above condition can be 
satisfied when the entries of the color map are 
preferably segmented into a continuous numerical 
range of indices for each of the predefined fea- 
tures. In general, this can be done so long as the 
features are either disjoint or related as sets and 
subsets. 

Since each index corresponds to a particular 
combination of features as well as a color, the 
particular feature associated with a specified pixel 
location can be found by reading the index from 
the pixel bit map and then finding the particular 
combination of features associated with the index. 

In accordance with an important aspect of the 
present invention, the memory of the video display 
system 10 further stores data describing the fea- 
tures associated with each index. Therefore, given 
the index of a user specified pixel location, the 
combination of features at that pixel location can be 
determined by reading from memory the descrip- 
tive data for the features associated with the index. 
In particular, as shown in FIG. 2 the random access 
memory 16 stores text strings 26 including a par- 
ticular string for each feature. Also, the random 
access memory 16 stores a text pointer table 27 



which specifies the text strings associated with 
each index. As further shown in FIG. 2, the text 
pointer table is very compact due to the fact that 
the text pointer table need merely store the nu- 

s merical limits of each range of indices for the 
segment associated with each feature. Moreover, 
the text pointer table is relatively compact because 
the indices are associated with pointers to the text 
strings rather than the strings themselves. 

10 By inspection of the contents of the color map 

25, the text strings 26, and the text pointer table 
27, it should be apparent that the first segment of 
the color map is addressed by indices ranging 
from 0 to 31 and it stores colors for the chair. The 

75 second segment of the color map stores colors at 
indices 32 to 63 for the lamp. Similarly, the third 
segment stores colors at indices 64 to 95 for 
displaying the sofa. 

As a specific exarriple, the user may operate 

20 the mouse 20 to move the cursor to a pixel on the 
chair having a light blue color represented by the 
red-green-blue video data triple of zero, zero, 31. 
The data processing unit 15 may operate the inter- 
face circuits 19 to receive the pixel address of that 

25 pixel. By addressing the pixel bit map with that 
pixel address, the data processing unit will read an 
index value of 1. Then, by comparing that index 
value to the indices in the text pointer table 27, the 
data processing unit will decide that it falls within 

30 the range of 0 to 31 specified for the pointer 
"STRING1." By addressing the text strings 26 
starting at the pointer STRING1 and continuing up 
to but not including the next string pointer 
STRING2, the computer may read and display the 

35 text "CHAIR" which describes the chair. 

Conversely, the user may enter the word 
"CHAIR" via the keyboard. The data processing 
unit 15 may obtain the word chair from the inter- 
face circuits 19, and search the text strings 27 for a 

40 match. The computer will then determine that the 
text entered by the user matches the text string 
starting at the address of "STRING1 " and continu- 
ing up to but not including the pointer "STRING2." 
The computer may then search the text pointer 

45 table for the occurrence of these pointers, and 
thereby determine that the feature associated with 
the first text string is specified for indices ranging 
from 0 to 31. Then, by temporarily changing the 
video data stored in the color map 25 at the indices 

so 0 to 31, the video display 11 will temporarily 
change the color of the chair to indicate to the user 
that the chair is a feature of the image responsive 
to the text that the user had entered on the key- 
board 18. 

55 Turning now to FIG. 3, there is shown a 

flowchart generally designated 30 of a control pro- 
cedure for performing the operations just de- 
scribed. In the first step 31, the data processing 
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unit 15 displays to the user a message requesting 
the user to select an image, it being understood 
that the disc memory 17 includes a plurality of 
images for selection by the user. In step 32 the 
data processing unit 15 operates the interface cir- 
cuits 19 to input the user's image selection which 
is to be entered via the keyboard 18. If the user 
presses the "escape" key on the keyboard 18, 
then in step 33 the data processing unit recognizes 
that the user desires to terminate or exit from the 
control procedure. Otherwise, the user's response 
should be information identifying a particular image 
to be viewed. Therefore, in step 34, the data pro- 
cessing unit transfers the record of the selected 
image from the disc memory 17 to the pixel bit 
map 13, the color map 25, and preassigned mem- 
ory locations in the random access memory 1 6 for 
the text pointer table 27 and the text strings 26. 

To determine the specific identification opera- 
tion desired by the user, in step 35 the data 
processing unit polls the keyboard by operating the 
interface circuits 19. As tested in step 36, if the 
user hits the "escape" key, execution jumps to 
step 31 to enable the user to select a new image. 
Otherwise, if text is received from the user, as 
tested in step 37, then in step 38 the data process- 
ing unit searches the text strings 26 for a match 
with the user specified text. If a match does not 
occur, as tested in step 39, then in step 40 a 
message is displayed to the user to indicate that a 
responsive feature cannot be found. Otherwise, the 
result of the match is a list of string pointers to the 
responsive text. In step 41, the data processing 
unit searches the text pointer table to determine 
the ranges of indices of the features corresponding 
to the string pointers. To display the responsive 
features to the user, the data processing unit tem- 
porarily changes the color in the color map for 
each index in the index range of the text pointer 
table specified for each matching string. This could 
be done a number of times to repetitively "flash" 
the responsive features. 

If in step 37 it was found that the user had not 
entered text, then in step 42 the data processing 
unit operates the interface circuits 19 to poll the 
pointing device 20. In step 43, the data processing 
unit tests whether the user is pointing to an image 
location. If so, then in step 44 the pixel bit map 13 
is read at the user specified pixel address to obtain 
a corresponding color index. Next, in step 45, the 
data processing unit searches the text pointer table 
for the text string pointer specified for the color 
index. One specific way of searching the text point- 
er table is to successively compare the index to 
each of the indices specified in the table. This 
method is illustrated by the computer program 
appended to the present specification. Once the 
text string pointer is obtained, then in step 46 the 



data processing unit reads the text string indicated 
by the text string pointer, and finally in step 47 the 
data processing unit displays the text string to the 
user. Execution then jumps back to step 35 to 
5 perform additional identification operations, if de- 
sired by the user. 

Turning now to FIG. 4, there is shown a 
flowchart generally designated 50 of a control pro- 
cedure for encoding an image to obtain the stored 

w data in the format illustrated in FIG. 2, In the first 
step 51, the user is told to input image data. In 
order to terminate the program when desired by 
the user, in step 52 the keyboard is polled and in 
step 53 execution terminates if the user hits the 

75 "escape" key. Otherwise, the user hits the key- 
board when the video data for an image is avail- 
able. The video data, for example, is provided by a 
video camera (not shown), an analog-to-digital con- 
verter (not shown), arid a frame buffer (not shown). 

20 In step 55 this video data is stored in the random 
access memory 16 in a pixel array of color values. 

In order to display the raw image, in step 56 
the distinct colors of the video data are transferred 
to a secondary color map area in random access 

25 memory 16. Also, in step 57 the secondary color 
map indices are loaded into a secondary pixel bit 
map set aside in the random access memory. In 
other words, the secondary pixel bit map and the 
secondary color map are set aside in order to 

30 construct a color mapped representation of the 
video image. In step 58, the color mapped video 
image is displayed by transferring the contents of 
the secondary color map and the secondary pixel 
bit map to the "primary" color map 25 and the 

35 "primary" pixel bit map 13. 

In order to encode information about predefin- 
ed features of the image, in step 59 a message is 
displayed telling the user to enter text describing a 
new feature. In step 60, the keyboard is polled to 

40 either receive the text or to receive an escape 
which would indicate that no more features are to 
be defined. This condition is tested in step 61. if 
text is entered, then in step 62 the text string is 
stored at the next available text string addresses, 

45 and the associated string pointers at the beginning 
of the string and just after the end of the string are 
stored in the text pointer table 27. Then, in step 63, 
the user is told to trace the outline of a new 
feature. To do this, the user operates the mouse 

so 20. In step 64, the data processing unit inputs the 
pixel coordinates of the outline, and in step 65 
computes the pixel coordinates bounded by the 
outline; these pixel coordinates have the same row 
addresses as the pixel coordinates of the outline 

55 and for each row address they include all of the 
column addresses within the column addresses of 
the pixels in the outline. Therefore, the location of 
the new feature has been specified by all of the 



7 



13 



EP 0 385 008 A1 



14 



pixels included in that feature. It should be noted, 
however, that a single feature may be comprised of 
entirely separate regions of pixels, in which case 
the user should outline each region included in the 
feature. 

Next, in step 66, the next available segment of 
the secondary color map indices is allocated or set 
aside to receive the colors of the new feature. In 
step 67, this new segment of the secondary color 
map is filled with the colors of the feature. Simi- 
larly, in step 68, the secondary pixel bit map at the 
pixel addresses of the feature are filled with the 
indices from the new segment of the secondary 
color map memory. Execution then jumps to step 
59 in order to define a new feature. Alternatively, 
the user may hit the "escape" key on the key- 
board, so that in step 69 the new image being built 
up in the secondary bit map and the secondary 
color map will be swapped with the contents of the 
primary memories and therefore displayed to the 
user. This interactive displaying could be useful for 
permitting the user to decide how many color map 
memory addresses should be allocated to each 
feature. The user could, for example, be provided 
with the option of interrupting the control procedure 
and going back and allocating more color map 
memory addresses to the previously defined fea- 
ture, or even for taking some color map memory 
addresses from some previously entered features 
and reallocating them to the feature that was just 
defined. In these cases of scarce color map mem- 
ory, the user could make available additional mem- 
ory addresses by eliminating certain colors from 
the features. 

In step 70 the data processing unit polls the 
keyboard to determine whether the user is ready to 
go on and define another feature or to terminate 
the program. In step 71 the data processing unit 
first responds by swapping the contents of the 
secondary pixel bit map and the secondary color 
map with the contents of the primary pixel bit map 
and the primary color map. Then, in step 72, 
execution jumps back to step 59 to define another 
feature unless the user has entered an "escape". If 
so, then all of the features have been defined for 
the image being encoded. However, at this time 
the secondary pixel bit map and the secondary 
color map do not include the information for defin- 
ing the background areas of the image which do 
not include any defined features. 

In order to provide information about the back- 
ground of the image, in step 73 the pixel coordi- 
nates of the background are computed or identified 
so that in step 74 the background colors can be 
loaded into the remaining entries of the secondary 
color map. Also, the pixel coordinates of the back- 
ground are computed or identified so that in step 
75 the secondary pixel bit map is filled at the 



background pixel addresses with indices of the 
background colors in the secondary color map. 
After this is -done, the secondary pixel bit map and 
the secondary color map include the entire en- 
5 coded image. Therefore, in step 76, the secondary 
maps, text and pointers can be stored in the disc 
memory 17 for use by the identification procedure 
of FIG. 3. 

One advantage of segmenting the color map 

10 memory is that the pixel bit map for a feature and 
the color map segment for the feature are sever- 
able from the composite encoded image. There- 
fore, it is possible to independently compose or 
remove, edit, and reinsert the video data for a 

75 feature of the composite encoded image. 

In view of the above, there has been described 
a video display terminal that allows a user to obtain 
descriptive information concerning a feature of a 
displayed image by pointing to the location of the 

20 feature. Conversely, the user may enter descriptive 
textual information, and the locations of responsive 
features are visually indicated. The digitized video 
images are efficiently stored along with textual in- 
formation about various features of the images and 

25 information about the locations of the features in 
the images. In particular, the information about the 
video image is encoded as a pixel bit map and a 
color map in which the addresses or indices of the 
color map are correlated with the addresses or 

30 pointers to descriptive information about predefined 
features of the video image. The correlation be- 
tween color map addresses and the descriptive 
information about the features is represented most 
compactly by arranging or sorting the entries in the 

35 colored map so that there is a correspondence 
between each predefined feature and a continuous 
range of color map addresses. Due to the high 
correlation in real-world images between color and 
features of interest, the memory capacity of the 

40 color map need not be substantially increased to 
provide the correlation between the physical loca- 
tions of the features and the descriptive information 
about the features. 

Although the invention has been described in 

45 connection with a specific embodiment that cor- 
relates textual information with the features of an 
image, the present invention is applicable to cor- 
relating other kinds of information about the defined 
features such as additional pictorial information for 

so showing the selected features in greater detail, or 
predefined computer programs or subroutines 
which relate to the defined features. It should be 
apparent to one of ordinary programming skill that 
the specific embodiment of FIGS. 2 and 3 could be 

55 readily modified to provide these capabilities by 
substituting for the text strings either strings of 
pixel information or strings of computer program 
steps. 
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To provide an "histology atlas," it would be 
desirable to use a pointer table (27 in FIG. 2) 
having a column of pixel string pointers as well as 
a column of text string pointers. When the user 
selects a feature of the displayed image, a mag- 
nified view of the selected feature could be dis- 
played along with the textual information describing 
the selected feature. Alternatively, a mouse (20 in 
FIG. 1) could be used having a pair of push-button 
switches, and either a magnified view of the se- 
lected feature, or the textual information about the 
selected feature, could be displayed depending 
upon whether the user activates one or the other of 
the push-button switches. 

Strings of computer program steps could be 
useful for providing the user with a menu of 
choices about the selected feature. These menu 
choices could select data input and control options 
as well as data display options. Upon selection of a 
feature, the control procedure of FIG. 3 in step 46 
would read the specified program string pointer 
from the string pointer table, transfer that pointer to 
the operand of an upcoming "jump" or "jump to 
subroutine" instruction, and then execute the 
"jump" or "jump to subroutine" instruction to there- 
by pass execution to the computer program steps 
specified for the selected feature. 

In general, computer program steps specified 
for predefined features would permit the program 
flow to be altered in any desired fashion, including 
alteration of the original procedure for displaying 
the image, and execution of data input and control 
procedures related to the selected feature. This 
would be most useful for hierarchical operation of 
an interactive computer terminal in which the 
strings of computer program steps specified for 
certain high-level features would include steps to 
interrupt the display of an original image, recur- 
sively display a magnified image of the selected 
feature, and redisplay the original image upon re- 
turning from the interrupt. Therefore, a user could 
successively select more detailed features until 
reaching the highest available level of magnification 
representing a very specific feature or control func- 
tion. 



Claims 

1. A video display system (10) for displaying 
an image (21) to a user and permitting the user to 
point to predefined features of the image (21) and 
obtain descriptive information about the specified 
features, said video display system (10) compris- 
ing, in combination: 

a color video display (11) for displaying text and 
images, said video display (11) including means 
(12, 14) responsive to video data for specifying the 



color which is displayed at respective pixel loca- 
tions on said image (21), 

manually-operated input means (20) for enabling 
the user to point to a specified pixel location, 

5 said display system (1.0) being characterised by 
the provision of a memory (16, 17) including means 
for storing data in an indexed data structure having 
data elements associated with respective indices 
having different values, said memory storing data 

10 including a color map (25) specifying video data for 
each of a plurality of indices, and a pixel bit map 
(1 3) specifying one of said indices for each of said 
pixel locations thereby defining the content of said 
image (21), said image (21) having a plurality of 

15 predefined features, said features occurring at cer- 
tain of said pixel locations, said pixel bit map (13) 
specifying for each of said features respective ones 
of said indices at said certain of said pixel loca- 
tions, said memory (16, 17) also storing respective 

20 strings (26) of descriptive information for said fea- 
tures, said strings (26) being specified for said 
respective ones of said indices, 
means (15) responsive to said input means (20) for 
reading said pixel bit map (13) to obtain the index 

25 specified for the specified pixel location, 

means (27) for reading the string (26) specified for 
said index specified for the specified pixel location, 
and 

means (15) for operating the video display (11) in 
30 response to the strings (26) so read. 

2. The video display system (1 0) as claimed in 
claim 1 , characterised in that said respective ones 
of said indices for each feature are included in 
continuous numerical ranges, the limits of said nu- 

35 merical ranges are stored in said memory (16, 17), 
and said means (27) for reading the string (26) 
includes means for comparing the limits to said 
index specified for the specified pixel location to 
determine the continuous numerical range includ- 

40 ing said index and thereby determine the feature 
indicated by the user, and means for reading the 
string (26) which describes the indicated feature. 

3. The video display system (10) as claimed in 
claim 1 or 2, characterised by the provision of 

45 further means (18) for receiving descriptive infor- 
mation from the user, means (15) for comparing 
the information received from the user to the de- 
scriptive strings (26) to select at least one descrip- 
tive string (26) which is responsive to the descrip- 

50 tive information received from the user, means (15) 
for reading the memory to determine the set of 
indices for which the selected string is specified, 
and means for changing the video data in said 
color map (25) that are specified for the indices in 

55 said set of indices, to thereby provide a means for 
visually indicating to the user the predefined fea- 
tures in the image which are responsive to the 
descriptive information received from the user. 
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4. The video display system (10) as claimed in 
any preceding claim, characterised in that said 
strings (26) are strings (26) of text, and said means 
(12, 14) for operating the video display (11) in- 
cludes means for displaying the string so read. 

5. The video display system (10) as claimed in 
any preceding claims, characterised in that said 
strings are strings of pixel information defining their 
respective features with increased magnification 
and resolution, and said means (12, 14) for operat- 
ing the video display (11) includes means for dis- 
playing the string of pixel information so read. 

6. The video display system (10) as claimed in 
claim 1 , 2, 3 or 4, characterised in that said strings 
are strings of computer program steps defining 
respective computer programs relating to the 
predefined features, and said means (12, 14) for 
operating the video display (11) includes means for 
executing the string of computer program steps so 
read. 

7. A method of operating a video display sys- 
tem (10) having a data processor (15), a color 
video display (11) including means (12, 14) respon- 
sive to video data for specifying the color which is 
displayed at respective pixel locations on an image . 
(21), a manually operated Input means (20) for 
specifying a pixel location, and a memory (16, 17) 
including means for storing data in an indexed data 
structure having data elements associated with re- 
spective indices having different values, said mem- 
ory storing data including a color map (25) specify- 
ing video data for each of a plurality of indices, a 
pixel bit map (13) specifying one of said indices for 
each of said pixel locations thereby defining the 
content of said image, said image (21) having a 
plurality of predefined features, said features occur- 
ring at certain of said pixel locations, said pixel bit 
map (13) specifying for each of said features re- 
spective ones of said indices at said certain of said 
pixel locations, said memory (16, 17) also storing 
respective strings (26) of descriptive text for said 
features, said text strings (26) being specified for 
said respective ones of said indices, said method 
characterised by the steps of: 

operating said input means (20) to receive an in- 
dication of a specified pixel location, 
reading said pixel bit map (13) to obtain the index 
specified for the specified pixel location, 
reading the text string (26) specified for said index 
specified for the specified pixel location, and 
displaying the text string (26) specified for the 
index specified for the specified pixel location. 

8. The method as claimed in claim 7, charac- 
terised in that said text strings (26) are specified for 
said respective ones of said indices by respective 
text string pointers (27) stored in said memory, and 
specified for said respective ones of said indices, 
and wherein said step of reading the text string 



(26) comprises the steps of reading the pointer (27) 
specified for the index specified for the specified 
pixel location, and reading the text string (26) in- 
dicated by the pointer (27) so read. 
5 9. A method of operating a data processor (15) 

having a memory (16, 17) in order to encode and 
store in said memory video data representing a 
color image (21) together with descriptive text 
about certain predefined features of said image 
10 (21 ) and the locations of said features in the image 
(21), said image (21) being subdivided into pixels 
at locations on said image, said feaures comprising 
certain predefined sets of said pixels, said memory 
(16, 17) including means for storing data in an 
75 indexed data structure having data elements asso- 
ciated with respective indices having different val* 
ues, said method characterised by the steps of; 
storing in said memory (16, 17) a color map (25) 
including at least one ehtry for each different color 
20 to be included in said image, and additional entries 
for each color occurring both inside and outside a 
predefined feature so that a unique entry is pro- 
vided for each color included in a distinct combina- 
tion of features, the color map (25) entries being 
25 addressable by respective indices, each index 
thereby having associated with it a predefined set 
of features and a certain color, 
storing in said memory (16, 17) data describing the 
features associated with each index, and 
30 storing in said memory (16, 17) a pixel bit map (13) 
specifying for each pixel the index for the combina- 
tion of features including that pixel and addressing 
the image color at the pixel location. 

10. A method of operating a video display 
35 system (20) having a data processor (15), a color 
video display (17) including means (12, 14) respon- 
sive to video data for specifying the color displayed 
at respective pixel locations on an image, means 
for receiving descriptive text from the user (18), 
40 and a memory (16, 17) including means for storing 
data in an indexed data structure having data ele- 
ments associated with respective indices having 
different values, said memory (16, 17) storing data 
including a color map (25) specifying video data for 
45 each of a plurality of indices, a pixel bit map (13) 
specifying one of said indices for each of said pixel 
locations thereby defining the content of said im- 
age, said image having a plurality of predefined 
features, said features occurring at certain of said 
so pixel locations, said pixel bit map (13) specifying 
for each of said features respective ones of said 
indices at said certain of said pixel locations, said 
memory (16, 17) also storing respective strings 
(26) of descriptive information for said features, 
55 said strings (26) being specified for said respective 
ones of said indices, 

said method characterised by the steps of: 
operating said means for receiving to receive de- 

10 



19 



BP 0 385 008 A1 



scriptive text from the user, 

comparing the text received from the user to the 
descriptive strings (26) to select at least one of the 
descriptive strings (26) which is responsive to the 
descriptive text received from the user, 5 
reading the memory to determine the set of indices 
for which the selected string (26) is specified, and 
changing the video data in said color map (25) that 
are specified for the indices in said set of indices, 
to thereby visually indicate to the user the predefin- 10 
ed features in the image which are responsive to 
the descriptive text received from the user. 
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